Apparatus for detecting inclination angle and controller

ABSTRACT

An apparatus for detecting an inclination angle includes a processor configured to input an image generated by a camera into a first classifier that has been trained to detect a two-wheeler, thereby detecting an object region including the two-wheeler in the image; rotate the object region by predetermined different angles to define characteristic regions as respective rotated regions; cut out the characteristic regions from the image; and detect a width of the two-wheeler in each of the characteristic regions and detects a rotation angle relative to the image as the inclination angle of the two-wheeler to the normal of the ground, based on rotation angles of the respective characteristic regions relative to the image and widths of the two-wheeler in the respective characteristic regions, the rotation angle minimizing the width of the two-wheeler.

FIELD

The present invention relates to an apparatus for detecting theinclination angle of a two-wheeler represented in an image to the normalof the ground, and a controller including such an apparatus to controltravel of a vehicle.

BACKGROUND

To predict the motion of a two-wheeler traveling near a host vehicle,techniques have been proposed to detect the inclination angle of thetwo-wheeler to the normal of the ground (e.g., see Japanese UnexaminedPatent Publications Nos. 2017-185862 and 2017-102928). In the presentapplication, the slope angle of the line joining the point where atwo-wheeler viewed from a position in front or behind touches the groundand the top of the head of its rider to the normal of the ground isreferred to as the inclination angle of the two-wheeler to the normal ofthe ground. This angle may be simply referred to as the “inclinationangle,” below.

For example, a drive support system disclosed in Japanese UnexaminedPatent Publication No. 2017-185862 detects the angle of bank of amotorcycle passing a host vehicle, and determines whether the motorcyclewill cut in on the host vehicle, based on the detected angle of bank.

A method for operating a vehicle disclosed in Japanese Unexamined PatentPublication No. 2017-102928 includes determining a mean column value ofpixels associated with a two-wheeler in at least two lines of an image,and determining the inclination angle of the two-wheeler, based on themean column values.

SUMMARY

Since a rider of a two-wheeler generally inclines the two-wheeler at achange of the direction of travel to the side to which he/she wishes totravel, the inclination angle of a two-wheeler is important informationto predict the motion thereof. Thus, it is desirable to correctly detectthe inclination angle of a two-wheeler represented in an image.

It is an object of the present invention to provide an apparatus thatcan detect the inclination angle of a two-wheeler represented in animage to the normal of the ground.

According to an embodiment, an apparatus for detecting an inclinationangle is provided. The apparatus includes a processor configured to:input an image generated by a camera into a first classifier that hasbeen trained to detect a two-wheeler, thereby detecting an object regionincluding the two-wheeler in the image; rotate the object region bypredetermined different angles to define characteristic regions asrespective rotated regions; cut out the characteristic regions from theimage; detect a width of the two-wheeler in each of the characteristicregions; and detect a rotation angle relative to the image as theinclination angle of the two-wheeler to the normal of the ground, basedon rotation angles of the respective characteristic regions relative tothe image and widths of the two-wheeler in the respective characteristicregions, the rotation angle minimizing the width of the two-wheeler.

The processor of the apparatus preferably resizes each of thecharacteristic regions to a predetermined size and inputs the resizedcharacteristic regions into a second classifier to determine thepositions of the left and right ends of the two-wheeler in each of thecharacteristic regions, and determines, for each of the characteristicregions, the difference between the positions of the left and right endsin the characteristic region as the width of the two-wheeler in thecharacteristic region, the second classifier having been trained todetect the positions of the left and right ends of the two-wheeler.

The processor of the apparatus preferably performs curve fitting usingpairs formed by the widths of the two-wheeler in the respectivecharacteristic regions and the rotation angles of the correspondingcharacteristic regions relative to the image to determine a fitted curverepresenting change in the width of the two-wheeler as a function ofchange in the rotation angle, and detects a rotation angle thatminimizes the width of the two-wheeler in the fitted curve to be theinclination angle of the two-wheeler to the normal of the ground.

According to another embodiment, an apparatus for detecting aninclination angle is provided. The apparatus includes a processorconfigured to: input an image generated by a camera into a firstclassifier that has been trained to detect a two-wheeler, therebydetecting an object region including the two-wheeler in the image;rotate the object region by predetermined different angles to definecharacteristic regions as respective rotated regions; cut out thecharacteristic regions from the image; detect a height of thetwo-wheeler in each of the characteristic regions; and detect a rotationangle relative to the image as the inclination angle of the two-wheelerto the normal of the ground, based on rotation angles of the respectivecharacteristic regions relative to the image and heights of thetwo-wheeler in the respective characteristic regions, the rotation anglemaximizing the height of the two-wheeler.

According to still another embodiment, a controller for controllingtravel of a vehicle is provided. The controller includes a processorconfigured to: input an image generated by a camera mounted on thevehicle into a first classifier that has been trained to detect atwo-wheeler, thereby detecting an object region including thetwo-wheeler located near the vehicle in the image; rotate the objectregion by predetermined different angles to define characteristicregions as respective rotated regions; cut out the characteristicregions from the image; detect a width of the two-wheeler in each of thecharacteristic regions; and detect a rotation angle relative to theimage as the inclination angle of the two-wheeler to the normal of theground, based on rotation angles of the respective characteristicregions relative to the image and widths of the two-wheeler in therespective characteristic regions, the rotation angle minimizing thewidth of the two-wheeler. The processor is further configured to predicta trajectory on which the two-wheeler will travel, based on the detectedinclination angle of the two-wheeler; determine a trajectory to betraveled of the vehicle, based on the predicted trajectory, so that thevehicle will be separated from the two-wheeler more than a predetermineddistance; and control the vehicle so that the vehicle will travel alongthe trajectory.

The apparatus according to the present invention has an advantageouseffect of being able to detect the inclination angle of a two-wheelerrepresented in an image to the normal of the ground.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 schematically illustrates the configuration of a vehicle controlsystem including the apparatus for detecting an inclination angle.

FIG. 2 illustrates the hardware configuration of an electronic controlunit, which is an embodiment of the apparatus.

FIG. 3 is a functional block diagram of a processor of the electroniccontrol unit, related to a vehicle control process including aninclination detecting process.

FIG. 4 illustrates an example of the configuration of a DNN used as thefirst classifier.

FIG. 5 illustrates an example of an object region and characteristicregions for a two-wheeler represented in an image.

FIG. 6 illustrates an example of the relationship between clippedcharacteristic regions and resized characteristic regions.

FIG. 7 illustrates an example of a detection result of the left andright ends of a two-wheeler obtained by the second classifier.

FIG. 8 is a diagram for briefly describing how an inclination angle isdetected using detection results of the width of the two-wheeler for therespective characteristic regions.

FIG. 9 is a timing chart of processes performed by the units related tothe inclination detecting process.

FIG. 10 is an operation flowchart of the vehicle control processincluding the inclination detecting process.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an apparatus for detecting an inclination angle will bedescribed with reference to the accompanying drawings. The apparatusdetects the inclination angle of a two-wheeler represented in an imageto the normal of the ground. To this end, the apparatus inputs the imageinto a first classifier that has been trained to detect a two-wheeler,thereby detecting a region including a two-wheeler represented in theimage. This region may be referred to as an “object region,” below. Theapparatus then rotates the object region by predetermined angles todefine characteristic regions as respective rotated regions, and cutsout the characteristic regions from the image. The apparatus thenresizes each of the characteristic regions to a certain size, and inputsinput data including individual pixel values of the resizedcharacteristic regions into a second classifier to determine, for eachof the characteristic regions, the width of the two-wheeler in thehorizontal direction of the characteristic region. This width may besimply referred to as the width of the two-wheeler in the characteristicregion, below. The apparatus then detects a rotation angle thatminimizes the width of the two-wheeler, as the inclination angle of thetwo-wheeler, based on the widths of the two-wheeler in the respectivecharacteristic regions and the rotation angles of the respectivecharacteristic regions relative to the original image.

The following describes an example in which the apparatus for detectingan inclination angle is applied to a vehicle control system. In thisexample, the apparatus performs an inclination detecting process onchronologically sequential images obtained by a camera mounted on avehicle to detect a two-wheeler near the vehicle, and detects theinclination angle of the detected two-wheeler. The vehicle controlsystem then predicts the motion of the two-wheeler, based on thedetected inclination angle, and performs automated driving control ofthe vehicle, depending on the predicted motion.

FIG. 1 schematically illustrates the configuration of a vehicle controlsystem including the apparatus for detecting an inclination angle. FIG.2 illustrates the hardware configuration of an electronic control unit,which is an embodiment of the apparatus. In the present embodiment, thevehicle control system 1, which is mounted on a vehicle 10 and controlsthe vehicle 10, includes a camera 2 for taking a picture of surroundingsof the vehicle 10, and an electronic control unit (ECU) 3, which is anexample of the apparatus. The camera 2 is connected to the ECU 3 so thatthey can communicate via an in-vehicle network 4 conforming to astandard, such as a controller area network. The vehicle control system1 may further include a storage device storing a map used for automateddriving control of the vehicle 10. The vehicle control system 1 mayfurther include a range sensor, such as LiDAR or radar; a receiver, suchas a GPS receiver, for determining the location of the vehicle 10 inconformity with a satellite positioning system; a wireless communicationterminal for wireless communication with another device; and anavigation device for searching for a planned travel route of thevehicle 10.

The camera 2 is an example of the image capturing unit that is a sensorfor detecting an object in a predetermined sensing range. The camera 2includes a two-dimensional detector constructed from an array ofoptoelectronic transducers, such as CCD or C-MOS, having sensitivity tovisible light and a focusing optical system focusing an image of atarget region on the two-dimensional detector. The camera 2 is mounted,for example, in the interior of the vehicle 10 so as to be oriented tothe front of the vehicle 10. The camera 2 takes a picture of a region infront of the vehicle 10 every predetermined capturing period (e.g., 1/30to 1/10 seconds), and generates images in which the front region isrepresented. The images obtained by the camera 2 are preferably colorimages. The vehicle 10 may include multiple cameras taking pictures indifferent orientations or having different focal lengths.

Every time generating an image, the camera 2 outputs the generated imageto the ECU 3 via the in-vehicle network 4.

The ECU 3 controls the vehicle 10. In the present embodiment, the ECU 3controls the vehicle 10 so as to automatically drive the vehicle 10,depending on an object detected from chronologically sequential imagesobtained by the camera 2. To this end, the ECU 3 includes acommunication interface 21, a memory 22, and a processor 23.

The communication interface 21, which is an example of a communicationunit, includes an interface circuit for connecting the ECU 3 to thein-vehicle network 4. In other words, the communication interface 21 isconnected to the camera 2 via the in-vehicle network 4. Every timereceiving an image from the camera 2, the communication interface 21passes the received image to the processor 23.

The memory 22, which is an example of a storing unit, includes, forexample, volatile and nonvolatile semiconductor memories. In the casethat the processor 23 includes multiple operation units as will bedescribed below, the memory 22 may include dedicated memory circuits forthe respective operation units. The memory 22 stores various types ofdata and various parameters used in the inclination detecting processperformed by the processor 23 of the ECU 3, e.g., images received fromthe camera 2, various parameters for specifying classifiers used in theinclination detecting process, and confidence-score thresholds forrespective types of objects. The memory 22 also stores various types ofdata generated in the inclination detecting process, such as adetected-object list indicating information related to detected objects,for a certain period. The memory 22 may further store information usedfor travel control of the vehicle 10, such as map information.

The processor 23, which is an example of a control unit, includes one ormore central processing units (CPUs) and a peripheral circuit thereof.The processor 23 may further include another arithmetic circuit, such asa logical operation unit, a numerical operation unit, or a graphicsprocessing unit (GPU). Every time receiving an image from the camera 2during travel of the vehicle 10, the processor 23 performs a vehiclecontrol process including the inclination detecting process on thereceived image. The processor 23 controls the vehicle 10 so as toautomatically drive the vehicle 10, depending on a detected object nearthe vehicle 10, in particular, a two-wheeler.

FIG. 3 is a functional block diagram of the processor 23 of the ECU 3,related to the vehicle control process including the inclinationdetecting process. The processor 23 includes an object detecting unit31, a characteristic-region defining unit 32, an inclination detectingunit 33, a tracking unit 34, a driving planning unit 35, and a vehiclecontrol unit 36. These units included in the processor 23 are, forexample, functional modules implemented by a computer program executedon the processor 23, or may be dedicated arithmetic circuits provided inthe processor 23. Of these units included in the processor 23, theobject detecting unit 31, the characteristic-region defining unit 32,and the inclination detecting unit 33 perform the inclination detectingprocess. In the case that the vehicle 10 includes multiple cameras, theprocessor 23 may perform, for each camera, the inclination detectingprocess, based on images obtained by the camera.

Every time receiving an image from the camera 2, the object detectingunit 31 inputs the latest received image into a first classifier forobject detection to detect an object region including a detectiontarget, such as a two-wheeler, represented in the image. Besidestwo-wheelers, examples of detection targets include objects that affecttravel control of the vehicle 10, e.g., other vehicles, humans,signposts, traffic lights, road markings such as lane division lines,and other objects on the road.

In the present embodiment, the object detecting unit 31 uses, as thefirst classifier, a deep neural network (DNN) that has been trained todetect an object region including a detection target (including atwo-wheeler) represented in an image and to identify the type of thedetection target. The DNN used by the object detecting unit 31 may be,for example, a DNN having a convolutional neural network (hereafter,“CNN”) architecture, such as a Single Shot MultiBox Detector (SSD) or aFaster R-CNN.

FIG. 4 illustrates an example of the configuration of a DNN used as thefirst classifier. The DNN 400 includes a main part 401, which isprovided on the input of images, and a position detecting part 402 and atype estimating part 403, which are provided closer to the output thanthe main part 401. The position detecting part 402 outputs an objectregion including a detection target represented in an image, dependingon the output from the main part 401. The type estimating part 403calculates confidence scores of respective types of detection targetsrepresented in object regions detected by the position detecting part402, depending on the output from the main part 401. The positiondetecting part 402 and the type estimating part 403 may be integrated.

The main part 401 may be, for example, a CNN including multiple layersconnected in series from the input toward the output. These multiplelayers include two or more convolution layers. The multiple layers ofthe main part 401 may further include a pooling layer every one or moreconvolution layers. The multiple layers of the main part 401 may furtherinclude one or more fully-connected layers. For example, the main part401 may be configured similarly to a base layer of an SSD.Alternatively, the main part 401 may be configured in accordance withanother CNN architecture, such as VGG-19, AlexNet, orNetwork-In-Network.

Upon input of an image, the main part 401 performs an operation on theimage in each layer to output a feature map calculated from the image.The main part 401 may output multiple feature maps of differentresolutions. For example, the main part 401 may output a feature mapwith the same resolution as the inputted image, and one or more featuremaps with a resolution lower than the inputted image.

The feature maps outputted from the main part 401 are inputted into theposition detecting part 402 and the type estimating part 403. Theposition detecting part 402 and the type estimating part 403 may be, forexample, CNNs each including multiple layers connected in series fromthe input toward the output. In the position detecting part 402 and thetype estimating part 403, the multiple layers of each CNN include two ormore convolution layers. In the position detecting part 402 and the typeestimating part 403, the multiple layers of each CNN may include apooling layer every one or more convolution layers. The convolutionlayers and the pooling layers may be common to the CNNs of the positiondetecting part 402 and the type estimating part 403. Additionally, inthe position detecting part 402 and the type estimating part 403, themultiple layers may include one or more fully-connected layers. In thiscase, the fully-connected layers are preferably provided closer to theoutput than the convolution layers. The outputs from the convolutionlayers may be directly inputted into the fully-connected layers. Theoutput layer of the type estimating part 403 may be a softmax layer thatcalculates confidence scores of respective types of detection targets inaccordance with a softmax function, or a sigmoid layer that calculatessuch confidence scores in accordance with a sigmoid function.

The position detecting part 402 and the type estimating part 403 aretrained so as to output confidence scores of respective types ofdetection targets, for example, for each of regions located at variouspositions in an image and having various sizes and aspect ratios. Thus,upon input of an image, the classifier 400 outputs confidence scores ofrespective types of detection targets for each of regions located atvarious positions in the image and having various sizes and aspectratios. The position detecting part 402 and the type estimating part 403then detect a region for which the confidence score of a certain type ofdetection target is not less than a predetermined confidence-scorethreshold, as an object region showing a detection target of this type.

Images (training images) included in training data used for training ofthe classifier 400 are tagged with, for example, types of detectiontargets (e.g., passenger vehicles, buses, trucks, and motorcycles) andcircumscribed rectangles of the detection targets, which are objectregions showing the detection targets.

The classifier 400 is trained with a large number of such trainingimages in accordance with a training technique, such as backpropagation.The use of the classifier 400 trained in this way allows the processor23 to accurately detect a target object, such as a two-wheeler, from animage.

The object detecting unit 31 may further perform a non-maximumsuppression (NMS) process to select one of object regions that areassumed to show the same object out of two or more overlapping objectregions.

The object detecting unit 31 enters, in a detected-object list, theposition and range of each object region in the image, and the type ofthe object included in the object region. The object detecting unit 31stores the detected-object list in the memory 22.

For each detected two-wheeler, the characteristic-region defining unit32 defines characteristic regions for determining the width of thetwo-wheeler. The following describes processing for a single two-wheelerbecause the characteristic-region defining unit 32 may perform the sameprocessing for each two-wheeler.

The characteristic-region defining unit 32 refers to the detected-objectlist to identify the object region showing the detected two-wheeler, androtates the object region by predetermined different angles relative tothe original image to define characteristic regions as respectiverotated regions. The characteristic-region defining unit 32 then cutsout the characteristic regions from the image. When two or moretwo-wheelers are entered in the detected-object list, thecharacteristic-region defining unit 32 may define the characteristicregions for each of the two-wheelers. The predetermined angles may be,for example, in units of 5°, 10°, or 20°.

FIG. 5 illustrates an example of an object region and characteristicregions for a two-wheeler represented in an image. In this example, arectangular object region 510 including a two-wheeler (together with arider) 501 represented in an image 500 is rotated by multiples of ±20°,and thereby five characteristic regions 510 to 514, which include theobject region 510 itself (i.e., the rotation angle is 0°) are defined.The characteristic-region defining unit 32 preferably defines thecharacteristic regions 510 to 514 so that the characteristic regions 510to 514 each cover the whole original object region 510. This preventspart of the left and right edges of the two-wheeler from being outside acharacteristic region, and allows a second classifier described below tocorrectly detect the positions of the left and right ends of thetwo-wheeler.

The characteristic-region defining unit 32 cuts out the definedcharacteristic regions from the original image. Each of the obtainedcharacteristic regions includes the two-wheeler that is a target fordetecting an inclination angle. Since the characteristic regions aredefined by rotating the corresponding object region by different angles,the slopes of the two-wheeler to the bottoms of the respectivecharacteristic regions are different. In general, the length of atwo-wheeler viewed from a position in front or behind is longer in thedirection of the line joining the point where the two-wheeler touchesthe ground and the top of the head of its rider (hereafter, referred toas the “center-line direction” for convenience of description) than inthe direction perpendicular to the center-line direction. For thisreason, as illustrated by the characteristic regions 510 to 514 in FIG.5 , the larger the angle formed between the bottom of a characteristicregion and the center-line direction of a two-wheeler, the smaller thewidth of the two-wheeler in the horizontal direction of thecharacteristic region (i.e., in the direction parallel to the bottom ofthe characteristic region). In other words, the smaller the differencebetween the rotation angle of a characteristic region relative to theoriginal image and the inclination angle of a two-wheeler, the smallerthe width of the two-wheeler in the horizontal direction of thecharacteristic region. Thus, detecting the widths of the two-wheelerfrom the respective characteristic regions enables estimation of theinclination angle of the two-wheeler.

The characteristic-region defining unit 32 performs scaletransformation, such as downsampling, upsampling, bi-linearinterpolation, or bi-cubic interpolation, to resize each of thecharacteristic regions cut out from the image so that the horizontal andvertical widths thereof may be a predetermined size (e.g., 64-by-64pixels). This results in characteristic regions of the same size beinginputted into a second classifier described below regardless of the sizeand the inclination angle of the two-wheeler in the image, and thussimplifies the configuration of the second classifier.

FIG. 6 illustrates an example of the relationship between characteristicregions cut out from an image and resized characteristic regions.Characteristic regions 610 to 614 in FIG. 6 , which correspond to thecharacteristic regions 510 to 514 in FIG. 5 , respectively, areunresized characteristic regions. Characteristic regions 620 to 624 areresized characteristic regions obtained by resizing the characteristicregions 610 to 614, respectively, to a predetermined size. The rotationangles of the unresized characteristic regions 610 to 614 relative tothe original image are different, and each of the characteristic regions610 to 614 is defined so as to cover the whole object region. Thus, thehorizontal and vertical sizes of the characteristic regions 610 to 614are different. However, the horizontal and vertical sizes of the resizedcharacteristic regions 620 to 624 are equal.

The characteristic-region defining unit 32 outputs the resizedcharacteristic regions to the inclination detecting unit 33. Thecharacteristic-region defining unit 32 also notifies the inclinationdetecting unit 33 of the ratio of the resized horizontal width to theunresized horizontal width (hereafter, referred to as the “normalizationratio” for convenience of description) for each of the characteristicregions.

The inclination detecting unit 33 detects the inclination angle of eachtwo-wheeler entered in the detected-object list. The following describesprocessing for a single two-wheeler because the inclination detectingunit 33 may perform the same processing for each two-wheeler.

The inclination detecting unit 33 inputs input data including individualpixel values of the resized characteristic regions into a secondclassifier that has been trained to detect the positions of the left andright ends of a two-wheeler, thereby detecting, for each of thecharacteristic regions, the positions of the left and right ends of thetwo-wheeler represented in the characteristic region. For each of thecharacteristic regions, the inclination detecting unit 33 detects thewidth of the two-wheeler represented in the characteristic region, basedon the positions of the left and right ends of the two-wheeler detectedfrom the characteristic region. The inclination detecting unit 33 thendetects a rotation angle that minimizes the width of the two-wheeler, asthe inclination angle of the two-wheeler, based on the widths of thetwo-wheeler in the respective characteristic regions and the rotationangles of the respective characteristic regions relative to the originalimage. In the present embodiment, the inclination detecting unit 33detects, as the inclination angle of the two-wheeler, that rotationangle of one of the characteristic regions relative to the originalimage which minimizes the width of the two-wheeler.

For example, assume that the size of the resized individualcharacteristic regions is 64-by-64 pixels, the number of characteristicregions is five, and the original image from which the characteristicregions are cut out is a color image represented by three channels ofRGB. Then, input data of a fourth-order tensor having a size of (5 [thenumber of characteristic regions], 3 [the number of channels], 64 [thenumber of pixels in the horizontal direction], 64 [the number of pixelsin the vertical direction]) is inputted into the second classifier.

The second classifier may be, for example, a neural network having a CNNarchitecture. In this case, the second classifier includes one or moreconvolution layers that perform a convolution operation for thehorizontal and vertical directions or for the horizontal, vertical, andchannel directions to generate a feature map; one or morefully-connected layers that perform a fully-connected operation on thefeature map calculated by the convolution layers; and an output layerthat outputs the positions of the left and right ends of the two-wheelerin each of the characteristic regions, based on the result of theoperation performed by the fully-connected layers. The second classifiermay further include, between the convolution layers, a pooling layerthat reduces the resolution of the feature map. The output layer uses,for example, a sigmoid function as an activation function, and outputsvalues indicating how likely the respective horizontal positions are tobe the left end of the two-wheeler in the range of [0,1] for each of thecharacteristic regions. In this case, for example, the position wherethe output value is 0 represents the left end of the two-wheeler, andthe position where the output value is 1 represents the right end of thetwo-wheeler. Alternatively, the output layer may regress the positionsof the left and right ends for each of the characteristic regionswithout using an activation function.

The second classifier has been trained in accordance withbackpropagation by using training data including pairs each formed by aset of resized characteristic regions like those described above andannotation information indicating the positions of the left and rightends of the two-wheeler in the characteristic regions included in thisset. The set of resized characteristic regions used as the training datais created, for example, by inputting an image for generating thetraining data into the first classifier to identify an object regionshowing a two-wheeler and then performing processing similar to that ofthe characteristic-region defining unit 32 on the identified objectregion.

FIG. 7 illustrates an example of a detection result of the left andright ends of a two-wheeler obtained by the second classifier. Asillustrated in FIG. 7 , the second classifier outputs the left endposition “xLeftNormalized” and the right end position “xRightNormalized”of a two-wheeler in a characteristic region 700 that has been resized,i.e., inputted into the second classifier.

For each of the resized characteristic regions, the inclinationdetecting unit 33 corrects the positions of the left and right ends ofthe two-wheeler outputted from the second classifier to the positions inthe unresized characteristic region, based on the normalization ratio ofthe characteristic region. More specifically, for each of the resizedcharacteristic regions, the inclination detecting unit 33 can divide thepositions of the left and right ends of the two-wheeler in thecharacteristic region by the normalization ratio of the characteristicregion to determine the positions of the left and right ends of thetwo-wheeler in the unresized characteristic region. Then, for each ofthe unresized characteristic regions, the inclination detecting unit 33calculates the difference between the positions of the left and rightends of the two-wheeler as the width of the two-wheeler in thecharacteristic region. The inclination detecting unit 33 identifies oneof the unresized characteristic regions that minimizes the width of thetwo-wheeler, and detects the rotation angle of the identifiedcharacteristic region relative to the original image as the inclinationangle of the two-wheeler.

For example, in the example illustrated in FIG. 5 , the width of thetwo-wheeler 501 is the smallest in the characteristic region 513, whichis obtained by rotating the object region clockwise by 40°, of the fivecharacteristic regions 510 to 514. Thus, the inclination detecting unit33 determines that the inclination angle of the two-wheeler 501 is 40°clockwise.

According to a modified example, the inclination detecting unit 33performs curve fitting using pairs formed by the widths of thetwo-wheeler and the rotation angles of the characteristic regions bymeans of a fitting technique, such as nonlinear least squares, toconstruct a fitted curve, such as a second-order curve. This fittedcurve represents change in the width of the two-wheeler as a function ofchange in the rotation angle relative to the original image. Theinclination detecting unit 33 may identify a rotation angle thatminimizes the width of the two-wheeler in the fitted curve, and detectthe identified rotation angle as the inclination angle of thetwo-wheeler.

FIG. 8 is a diagram for briefly describing how an inclination angle isdetected using detection results of the width of the two-wheeler for therespective characteristic regions according to this modified example.The abscissa in FIG. 8 represents the rotation angle of eachcharacteristic region relative to the original image. In this example,the rotation angle of a characteristic region rotated clockwise isnegative, and that of a characteristic region rotated anticlockwise ispositive. The ordinate represents the width of the two-wheeler. A curve800 represents a fitted curve for pairs 801 formed by the widths of thetwo-wheeler obtained for the respective characteristic regions and therotation angles of the corresponding characteristic regions. In thefitted curve 800 of this example, the width of the two-wheeler is thesmallest at a 40° rotation angle. Thus, the inclination detecting unit33 determines that the inclination angle of the two-wheeler is 40°.

According to this modified example, even when none of the rotationangles of the characteristic regions match the inclination angle of atwo-wheeler, the inclination detecting unit 33 can detect theinclination angle of the two-wheeler correctly.

For each two-wheeler entered in the detected-object list, theinclination detecting unit 33 stores, in the memory 22, the detectedinclination angle of the two-wheeler in association with information ofthe two-wheeler.

FIG. 9 is a timing chart of processes performed by the units related tothe inclination detecting process. The processes of the units of theprocessor 23 are managed, for example, by a scheduler (not illustrated)executed on the processor 23, and are performed in accordance with thetiming chart illustrated in FIG. 9 . The abscissa of FIG. 9 representstime. In FIG. 9 , individual blocks indicate execution of the processesshown in the respective blocks, and individual arrows indicate deliveryof data (e.g., images, the detected-object list, and characteristicregions) between the processes. For example, the ECU 3 receives an imagefrom the camera 2 at time t1, and then, the GPU included in theprocessor 23 performs a target-object detecting process of the objectdetecting unit 31 on the image. Before the target-object detectingprocess, preprocessing, such as contrast correction or color conversion,may be performed on the image.

After the target-object detecting process, the CPU included in theprocessor 23 performs postprocessing of object detection, such as entryof the types and the object regions of detected objects into thedetected-object list, and thereafter performs a tracking process of thetracking unit 34 described below. In parallel with the tracking process,the CPU performs a characteristic-region defining process of thecharacteristic-region defining unit 32 for a detected two-wheeler. Afterthe characteristic-region defining process, the GPU performs resizing ofthe individual characteristic regions, generation of that data to beinputted into the second classifier of the inclination detecting unit 33which includes the resized individual characteristic regions, and aprocess of the second classifier to detect the left and right ends ofthe two-wheeler in the characteristic regions. Based on this detectionresult, the CPU performs detection of the inclination angle of thetwo-wheeler. The result of the tracking process and that of detection ofthe inclination angle are used for the processes of the driving planningunit 35 and the vehicle control unit 36. Resizing of the individualcharacteristic regions, generation of data to be inputted into thesecond classifier, and the process of the second classifier to detectthe left and right ends of the two-wheeler in the characteristic regionsrotated by different rotation angles may be performed in batchprocessing.

The tracking unit 34 refers to the detected-object list to associate,for each object region detected from the latest image, the detectiontarget represented in the object region with a detection target detectedfrom a past image, thereby tracking the detection target represented inthe object region.

The tracking unit 34 applies, for example, a tracking process based onoptical flow, such as the Lucas-Kanade method, to the object region ofinterest in the latest image and the object regions in the past images,thereby tracking the detection target represented in the object regions.To this end, the tracking unit 34 applies, for example, a filter forextracting characteristic points, such as a SIFT or Harris operator, tothe object region of interest, thereby extracting multiplecharacteristic points from the object region. Then, the tracking unit 34may identify those points in the object regions in the past images whichcorrespond to each of the characteristic points in accordance with theapplied tracking technique, thereby calculating the optical flow.Alternatively, the tracking unit 34 may apply another trackingtechnique, which is applied for tracking a moving object detected froman image, to the object region of interest in the latest image and theobject regions in the past images, thereby tracking the detection targetrepresented in the object regions.

The tracking unit 34 regards a detection target that is detected fromthe latest image and associated with none of detection targetsrepresented in the past images as a new tracking target, assigns thisdetection target an identification number different from theidentification numbers of the other tracked detection targets, andenters the assigned identification number in the detected-object list.In contrast, the tracking unit 34 associates a detection target that isdetected from the latest image and associated with a detection targetrepresented in the past images, i.e., one of the tracked detectiontargets, with the same identification number as assigned to this trackeddetection target.

The driving planning unit 35 refers to the detected-object list togenerate one or more trajectories to be traveled of the vehicle 10 sothat the vehicle 10 will not collide with an object near the vehicle 10.Each trajectory to be traveled is represented as, for example, a set oftarget locations of the vehicle 10 at points in time from the currenttime to a predetermined time ahead thereof. For example, the drivingplanning unit 35 refers to the detected-object list to perform viewpointtransformation, using information such as the position at which thecamera 2 is mounted on the vehicle 10, thereby transforming the imagecoordinates of the objects in the detected-object list into coordinatesin an aerial image (“aerial-image coordinates”). The driving planningunit 35 then performs a tracking process on sequential aerial-imagecoordinates, using the Kalman filter, the Particle filter, or anotherfilter, to track the objects entered in the detected-object list, anduses the trajectories obtained from the tracking results to determinepredicted trajectories of the respective objects to a predetermined timeahead. When the detection target of interest is a two-wheeler, thedriving planning unit 35 uses the inclination angles of the two-wheelercorresponding to the times of acquisition of the respective imagesduring tracking to determine the predicted trajectory. For example,assume that the two-wheeler that is the detection target of interest isinclined anticlockwise relative to the normal of the ground (i.e.,inclined to the left as viewed from the vehicle 10). Then, the largerthe anticlockwise inclination angle is and becomes with the passage oftime, the higher the possibility that the two-wheeler will make a lanechange to the left or a left turn. Thus, the driving planning unit 35determines a predicted trajectory of this two-wheeler such that thelarger the anticlockwise inclination angle is and becomes with thepassage of time, the faster the two-wheeler makes a lane change to theleft or a left turn. Similarly, assume that the two-wheeler that is thedetection target of interest is inclined clockwise relative to thenormal of the ground (i.e., inclined to the right as viewed from thevehicle 10). Then, the larger the clockwise inclination angle is andbecomes with the passage of time, the higher the possibility that thetwo-wheeler will make a lane change to the right or a right turn. Thus,the driving planning unit 35 determines a predicted trajectory of thistwo-wheeler such that the larger the clockwise inclination angle is andbecomes with the passage of time, the faster the two-wheeler makes alane change to the right or a right turn. When the absolute value of theinclination angle of the two-wheeler that is the detection target ofinterest is not greater than a predetermined angle (e.g., 5°), it ishighly likely that the two-wheeler will travel straight. Thus, thedriving planning unit 35 determines a predicted trajectory of thistwo-wheeler such that it will travel straight.

The driving planning unit 35 generates a trajectory to be traveled ofthe vehicle 10, based on the predicted trajectories of the trackedobjects, and the location, speed, and orientation of the vehicle 10, sothat a predicted distance between the vehicle 10 and any of the trackedobjects will be not less than a predetermined distance until apredetermined time ahead. The driving planning unit 35 can estimate thelocation, speed, and orientation of the vehicle 10, based on, forexample, current location information that is obtained from a GPSreceiver (not illustrated) mounted on the vehicle 10 and indicates thecurrent location of the vehicle 10. Alternatively, every time an imageis obtained by the camera 2, a localizing processing unit (not shown)may detect lane division lines on the right and left of the vehicle 10from the image, and compare the detected lane division lines with themap information stored in the memory 22, thereby estimating thelocation, speed and orientation of the vehicle 10. Additionally, thedriving planning unit 35 may refer to, for example, the current locationinformation of the vehicle 10 and the map information stored in thememory 22 to count the number of lanes available for travel by thevehicle 10. When more than one lane is available for travel by thevehicle 10, the driving planning unit 35 may generate a trajectory to betraveled so that the vehicle 10 will make a lane change.

The driving planning unit 35 may generate multiple trajectories to betraveled. In this case, the driving planning unit 35 may select one ofthe torajectories such that the sum of the absolute values ofacceleration of the vehicle 10 is the smallest.

The driving planning unit 35 notifies the vehicle control unit 36 of thegenerated trajectory to be traveled.

The vehicle control unit 36 controls the components of the vehicle 10 sothat the vehicle 10 will travel along the notified trajectory to betraveled. For example, the vehicle control unit 36 determines theacceleration of the vehicle 10 in accordance with the notifiedtrajectory to be traveled and the current speed of the vehicle 10measured by a vehicle speed sensor (not illustrated), and determines thedegree of accelerator opening or the amount of braking so that theacceleration of the vehicle 10 will be equal to the determinedacceleration. The vehicle control unit 36 then determines the amount offuel injection in accordance with the determined degree of acceleratoropening, and outputs a control signal depending on the amount of fuelinjection to a fuel injector of the engine of the vehicle 10.Alternatively, the vehicle control unit 36 outputs a control signaldepending on the determined amount of braking to the brake of thevehicle 10.

When the vehicle 10 changes its course in order to travel along thetrajectory to be traveled, the vehicle control unit 36 determines thesteering angle of the vehicle 10 in accordance with the trajectory to betraveled, and outputs a control signal depending on the steering angleto an actuator (not illustrated) controlling the steering wheel of thevehicle 10.

FIG. 10 is an operation flowchart of the vehicle control process thatincludes the inclination detecting process and is performed by theprocessor 23. Every time receiving an image from the camera 2, theprocessor 23 performs the vehicle control process in accordance with theoperation flowchart illustrated in FIG. 10 . In the following operationflowchart, the process of steps S101 to S105 corresponds to theinclination detecting process.

The object detecting unit 31 of the processor 23 inputs the latest imageobtained from the camera 2 into the first classifier to detect detectiontargets represented in the image. More specifically, the objectdetecting unit 31 detects rectangular object regions including detectiontargets in the image (step S101). In addition, the object detecting unit31 identifies the types of the detected objects, and enters the detectedobjects in the detected-object list.

The characteristic-region defining unit 32 of the processor 23 rotatesan object region including a two-wheeler that is a detection targetdetected from the latest image by predetermined different anglesrelative to the original image to define characteristic regions, andcuts out the characteristic regions from the image (step S102). Inaddition, the characteristic-region defining unit 32 resizes each of thecharacteristic regions to a predetermined size (step S103).

Regarding the two-wheeler that is a detection target detected from thelatest image, the inclination detecting unit 33 of the processor 23inputs input data including individual pixel values of the resizedcharacteristic regions into the second classifier to detect thepositions of the left and right ends of the two-wheeler for each of thecharacteristic regions (step S104). The inclination detecting unit 33then determines the width of the two-wheeler from the positions of theleft and right ends of the two-wheeler for each of the characteristicregions, and detects, as the inclination angle of the two-wheeler, thatrotation angle of a characteristic region relative to the original imagewhich minimizes the width (step S105). As in the above-describedmodified example, the inclination detecting unit 33 may construct afitted curve representing change in the width of the two-wheeler as afunction of change in the rotation angle, based on pairs formed by thewidths of the two-wheeler in the respective characteristic regions andthe rotation angles of the corresponding characteristic regions relativeto the original image, and detect a rotation angle that minimizes thewidth of the two-wheeler in the fitted curve as the inclination angle ofthe two-wheeler.

For each of the object regions including the detection targets in thelatest image, the tracking unit 34 of the processor 23 tracks thedetection target represented in the object region of the latest image,based on this object region and the object regions in the past images(step S106).

The driving planning unit 35 of the processor 23 refers to thedetected-object list to generate a trajectory to be traveled of thevehicle 10 so that, for each of the detection targets entered in thedetected-object list, the trajectory to be traveled will be separatedmore than a predetermined distance from the predicted trajectory of thedetection target determined from the tracking result of the detectiontarget (step S107). When the detection target is a two-wheeler, thedriving planning unit 35 uses the inclination angle of the two-wheelerfor determining the predicted trajectory as described above. The vehiclecontrol unit 36 of the processor 23 then controls the vehicle 10 so thatthe vehicle 10 will travel along the trajectory to be traveled (stepS108). The processor 23 then terminates the vehicle control process.

As has been described above, the apparatus for detecting an inclinationangle inputs an image into a first classifier that has been trained todetect a two-wheeler, thereby detecting an object region including atwo-wheeler represented in the image. The apparatus then rotates theobject region by predetermined angles to define characteristic regionsas respective rotated regions, and cuts out the characteristic regionsfrom the image. The apparatus then resizes each of the characteristicregions to a predetermined size, and inputs input data includingindividual pixel values of the characteristic regions into a secondclassifier to determine the widths of the two-wheeler in the respectivecharacteristic region. The apparatus then detects, as the inclinationangle of the two-wheeler, that rotation angle of a characteristic regionwhich minimizes the width of the two-wheeler. In particular, theapparatus uses, as the second classifier, a classifier that has beentrained to detect the positions of the left and right ends of thetwo-wheeler in each of the resized characteristic regions, which allowsfor correctly detecting the widths of the two-wheeler in the respectivecharacteristic regions. The width of the two-wheeler in each of thecharacteristic regions changes as a function of the inclination angle ofthe two-wheeler to the bottom of the characteristic region, and thatrotation angle of a characteristic region relative to the original imagewhich minimizes the width of the two-wheeler is substantially equal tothe inclination angle of the two-wheeler to the ground. For this reason,the apparatus can correctly detect the inclination angle of atwo-wheeler by determining the inclination angle from that rotationangle of a characteristic region relative to the original image whichminimizes the width of the two-wheeler.

With reference to FIG. 5 again, it is assumed that the larger the angleformed between the bottom of a characteristic region and the center-linedirection of a two-wheeler, the larger the height of the two-wheeler inthe vertical direction of the characteristic region (i.e., in thedirection perpendicular to the bottom of the characteristic region).Thus, according to a modified example, the inclination detecting unit 33may detect the height of the two-wheeler from the positions of the topand bottom of the two-wheeler for each of the characteristic regions,and detect, as the inclination angle of the two-wheeler, that rotationangle of a characteristic region relative to the original image whichmaximizes the height. In this case, as in the above-describedembodiment, the inclination detecting unit 33 may input input dataincluding individual pixel values of the resized characteristic regionsinto a second classifier that has been trained to detect the positionsof the top and bottom of a two-wheeler, thereby detecting, for each ofthe characteristic regions, the positions of the top and bottom of thetwo-wheeler represented in the characteristic region.

According to another modified example, the object detecting unit 31 mayuse a classifier other than a DNN, to detect a detection target from animage. For example, the object detecting unit 31 may use, as the firstclassifier, a support vector machine (SVM) that has been trained tooutput a confidence score indicating how likely a target object fordetection is to be represented in a window defined on an image, inresponse to input of features (e.g., HOG) calculated with respect to thewindow. The object detecting unit 31 calculates the features withrespect to a window defined on an image while variously changing theposition, size, and aspect ratio of the window, and inputs thecalculated features into the SVM to obtain the confidence score for thewindow. Then, the object detecting unit 31 may determine that a windowfor which the confidence score of a certain type of detection target isnot less than a predetermined confidence-score threshold shows thedetection target, and regard this window as an object region. The SVMmay be prepared for each type of target object for detection. In thiscase, the object detecting unit 31 may input, for each window, thefeatures calculated from the window into the SVMs to calculate theconfidence scores for the respective types of objects.

According to still another modified example, the input data to beinputted into the second classifier may include those features of thefeature maps outputted from the main part of the first classifier usedby the object detecting unit 31 which correspond to the positions ofindividual pixels of the resized characteristic regions.

The apparatus for detecting an inclination angle according to theembodiment or modified examples may be mounted on a device other thanvehicle-mounted equipment. For example, the apparatus according to theembodiment or modified examples may be configured to detect atwo-wheeler from an image generated by a surveillance camera placed fortaking a picture of a predetermined outdoor region every predeterminedperiod and to detect the inclination angle of the detected two-wheeler.

A computer program for achieving the functions of the units of theprocessor 23 of the apparatus according to the embodiment or modifiedexamples may be provided in a form recorded on a computer-readable andportable medium, such as a semiconductor memory, a magnetic recordingmedium, or an optical recording medium.

As described above, those skilled in the art may make variousmodifications according to embodiments within the scope of the presentinvention.

What is claimed is:
 1. An apparatus for detecting an inclination anglecomprising: a processor configured to: input an image generated by acamera into a first classifier that has been trained to detect atwo-wheeler, thereby detecting an object region including thetwo-wheeler in the image; rotate the object region by a plurality ofpredetermined different angles to define a plurality of characteristicregions as respective rotated regions; cut out the characteristicregions from the image; detect an image width of the two-wheeler in eachof the characteristic regions; detect a rotation angle relative to theimage as the inclination angle of the two-wheeler to the normal of theground, based on rotation angles of the respective characteristicregions relative to the image and image widths of the two-wheeler in therespective characteristic regions, the rotation angle minimizing theimage width of the two-wheeler; and control travel of a vehicle based onthe detected rotation angle; wherein the processor is further configuredto resize each of the characteristic regions to a predetermined size andinput the resized characteristic regions into a second classifier todetermine the positions of the left and right ends of the two-wheeler ineach of the characteristic regions, and determine, for each of thecharacteristic regions, the difference between the positions of the leftand right ends in the characteristic region as the image width of thetwo-wheeler in the characteristic region, the second classifier havingbeen trained to detect the positions of the left and right ends of thetwo-wheeler.
 2. An apparatus for detecting an inclination anglecomprising: a processor configured to: input an image generated by acamera into a first classifier that has been trained to detect atwo-wheeler, thereby detecting an object region including thetwo-wheeler in the image; rotate the object region by a plurality ofpredetermined different angles to define a plurality of characteristicregions as respective rotated regions; cut out the characteristicregions from the image; detect an image width of the two-wheeler in eachof the characteristic regions; detect a rotation angle relative to theimage as the inclination angle of the two-wheeler to the normal of theground, based on rotation angles of the respective characteristicregions relative to the image and image widths of the two-wheeler in therespective characteristic regions, the rotation angle minimizing theimage width of the two-wheeler; and control travel of a vehicle based onthe detected rotation angle; wherein the processor is further configuredto perform curve fitting using pairs formed by the image widths of thetwo-wheeler in the respective characteristic regions and the rotationangles of the corresponding characteristic regions relative to the imageto determine a fitted curve representing change in the image width ofthe two-wheeler as a function of change in the rotation angle, anddetect a rotation angle that minimizes the image width of thetwo-wheeler in the fitted curve to be the inclination angle of thetwo-wheeler to the normal of the ground.
 3. An apparatus for detectingan inclination angle comprising: a processor configured to: input animage generated by a camera into a first classifier that has beentrained to detect a two-wheeler, thereby detecting an object regionincluding the two-wheeler in the image; rotate the object region by aplurality of predetermined different angles to define a plurality ofcharacteristic regions as respective rotated regions; cut out thecharacteristic regions from the image; detect an image height and animage width of the two-wheeler in each of the characteristic regions;detect a rotation angle relative to the image as the inclination angleof the two-wheeler to the normal of the ground, based on rotation anglesof the respective characteristic regions relative to the image, theimage width of the two-wheeler, and image heights of the two-wheeler inthe respective characteristic regions, the rotation angle maximizing theimage height of the two-wheeler and minimizing the image width of thetwo-wheeler; predict a trajectory on which the two-wheeler will travel,based on the inclination angle; determine a trajectory to be traveled bya vehicle, based on the predicted trajectory, so that the vehicle willbe separated from the two-wheeler more than a predetermined distance;and control travel of the vehicle so that the vehicle will travel alongthe trajectory to be traveled.
 4. A controller for controlling travel ofa vehicle, the controller comprising: a processor configured to: inputan image generated by a camera mounted on the vehicle into a firstclassifier that has been trained to detect a two-wheeler, therebydetecting an object region including the two-wheeler located nearest thevehicle in the image; rotate the object region by a plurality ofpredetermined different angles to define a plurality of characteristicregions as respective rotated regions; cut out the characteristicregions from the image; detect an image width of the two-wheeler in eachof the characteristic regions and detects a rotation angle relative tothe image as the inclination angle of the two-wheeler to the normal ofthe ground, based on rotation angles of the respective characteristicregions relative to the image and image widths of the two-wheeler in therespective characteristic regions, the rotation angle minimizing theimage width of the two-wheeler; predict a trajectory on which thetwo-wheeler will travel, based on the inclination angle; determine atrajectory to be traveled by the vehicle, based on the predictedtrajectory, so that the vehicle will be separated from the two-wheelermore than a predetermined distance; and control the vehicle so that thevehicle will travel along the trajectory to be traveled.